Automating the Math
Automating the Math
Transcribing with MathJax and SRE
Background: Initial Problem

- No reliable, fast, and inexpensive way of producing Braille versions of mathematics texts.
- Several tools will produce convincing-looking, but incorrect output.
- Situation is particular bad for graduate texts
- And getting worse for research papers, etc.
Overview
- Math document formats and their accessibility
- PreTeXt and its accessibilty
- Accessible Math with MathJax/SRE
- Creating Tactile Hardcopies
- From PreTeXt to Tactile
- Nemeth with MathJax and Speech Rule Engine
- A word on graphics with screen reading
Document formats with math support
Accessibility of these formats
- Word formats
- Raw: Pretty good accessibility in editor, Math depends on
implementation/screen reader/formula editor!
- As PDF: Good structural accessibility, but no math
- LaTeX needs to be compiled into PDF
- Although LaTeX provides great structure that is lost in PDF
- Math should be good but is lost too!
- Markup like Markdown/ HTML
- Good structural accessibility
- Math is accessible if handled right
Best option is often to move content to the web
Conversion towards web formats
- pandoc - bi-directional "swiss army knife"
- markdown, TeX, docx, and many many more
- LaTeX
- PDF
Keep as much of the source as possible: do not replace the LaTeX
Why is LaTeX important
- LaTeX notation is lingua franca in Math
- Also usable in informal communication like email
- Can be used on the web (github, stackoverflow, etc.)
- Every new Math editor ultimately allows LaTeX as input syntax
- LaTeX for the visually impaired
- Some European countries teach LaTeX to VI stduents from grade 1
- Little additional effort to learn
- Easy to communicate with non-specailist teachers and sighted peers
- Translation of code to 8-dot LaTeX Braille
Creating Tactile Hardcopies
Translate textbooks automatically into Braille from source
- Little support in for most formats
- LaTeX package do not produce quality results
What do we do?
- Sources in PreTeXt
- Text into literary Braille with LibLouis
- Math into Nemeth with MathJax and Speech Rule Engine
- Graphics into tactile graphics via a multistep pipeline
What is PreTeXt?
PreTeXt is a an uncomplicated XML vocabulary for
open source textbooks, monographs, and research articles.
Aims to bring together the good parts of HTML/Word with LaTeX.
- Structure in XML
- Mathematics in LaTeX
- Graphics mainly in LaTeX
- Over 100 individual projects:
- mainly Math, some CS and engineering
- one music theory, one university writing handbook
In our context: provides free and well curated content
PreTeXt and Accessibility
- One source, many outputs:
- print, PDF, web, EPUB, Jupyter Notebooks, ...
- HTML output is accessible
- Mathematics is in LaTeX
- Rendered with MathJax
- Made accessible using Speech Rule Engine
What is MathJax?
- MathJax is an open-source JavaScript library for
rendering Mathematics in all browsers
- Can take {\LaTeX}, AsciiMath, and MathML as input
- Generates browser output, e.g. HTML/CSS, SVG
- MathJax is the de facto rendering solution of most Mathematics on the web
- Standard Maths rendering solution for many publishers
- Around for over 10 years, lastest version 3.1 released Aug 2020
In our context: translation of LaTeX to Nemeth Braille
Speech Rule Engine
- Javascript library for translating Math into Speech available at github
- Only Math speech solution in JavaScript: Runs in browser, command line, as node module.
- Speech solution for: ChromeVox, MathJax, EquatIO, MathLive, ...
- Based on semantic interpretations via pattern matching techniques
- Ignores any poor syntax (or at least tries to)
- Math markup (e.g., LaTeX, MathML) are not very expressive, sometimes bloated
SRE in MathJax
$$x = {-b \pm \sqrt{b^2-4ac} \over 2a}$$
- SRE is pulled into MathJax
- Speech (or Braille) with
- Interactive Exploration
- Highlighting, magnification, etc.
- All this is done using a semantic interpretation
Generate Semantics
ax^2+bx+c=0
<math>
<mi>a</mi>
<msup>
<mi>x</mi>
<mn>2</mn>
</msup>
<mo>+</mo>
<mi>b</mi>
<mi>x</mi>
<mo>+</mo>
<mi>c</mi>
<mo>=</mo>
<mn>0</mn>
</math>
Generate Speech
$$x = {-b \pm \sqrt{b^2-4ac} \over 2a}$$
Speech
Play Area for speech
- Multiple reading rules and preferences that can be swapped
- MathSpeak, ClearSpeak, (ChromeVox)
- Summarisation, Navigation info
- 12 different locales
- English, German, Spanish, French, Italian, ...
- Tactile formats
- 8-dot LaTeX Braille (German, Slovenian, ...)
- Nemeth (linear)
- Nemeth 2D
Nemeth in SRE
Sponsored by Big Ten Academic Alliance
- Once semantics is embedded speech and other alternative formats can be easily generated
- Tactile output, Abstraction, Summaries
$$ \vec{\nabla} \times \vec{F} =
\left( \frac{\partial F_z}{\partial y} - \frac{\partial F_y}{\partial z} \right) \mathbf{i}
+ \left( \frac{\partial F_x}{\partial z} - \frac{\partial F_z}{\partial x} \right) \mathbf{j}
+ \left( \frac{\partial F_y}{\partial x} - \frac{\partial F_x}{\partial y} \right) \mathbf{k}
$$
Click or focus on the formula and press ENTER to start, Escape to quit.
MathJax on the Server
- work on the server as easily as in the browser
- works in non-standard DOMs (e.g., markdown or PreTeXt)
- separable pipeline for
- finding
- converting
- annotating
- rendering math
- E.g., LaTeX in a document can be translated into SVG with speech or Braille annotation.
2D in Nemeth
- Nemeth for embossing requires
- Large fractions
- Matrices, vectors
- Equation systems
- Cayley tables
- SRE can generate 2D output
- Uses same rules as for Braille display output during screen reading
- Applies abstract styling rules
- Some sample output
Quality Assurance
- We developed a Webapp for Nemeth Translation
- Basic idea:
- Proof read and correct automatically translated math
- Easy reference to context via links source
- Selection of semantically unique samples vs full list of expressions
Some Challenging Examples
Nemeth is challenging to get right automatically, but sometimes automation is better:
- Subtlety of spaces:
- $44,352,000$ is a single number and transcribes as ⠼⠲⠲⠒⠢⠆⠴⠴⠴
- $(0110,1110,0110)$ is not a single number but a vector, hence ⠷⠼⠴⠂⠂⠴⠀⠼⠂⠂⠂⠴⠀⠼⠴⠂⠂⠴⠾
- $(n, E) = (451{,}231)$ The right hand side is not a single number ⠷⠝⠠⠀⠠⠑⠾⠀⠨⠅⠀⠷⠲⠢⠂⠠⠀⠆⠒⠂⠾
- Context helps
$$PAP^{-1} = B$$
is not an abbreviation: ⠠⠏⠠⠁⠠⠏⠘⠤⠂⠀⠨⠅⠀⠠⠃
Voicing Complex Content
$$
\require{bussproofs}
\begin{prooftree}
\AxiomC{}
\RightLabel{$Hyp^{1}$}
\UnaryInfC{$P$}
\AXC{$P\to Q$}
\RL{$\to_E$}
\BIC{$Q^2$}
\AXC{$Q\to R$}
\RL{$\to_E$}
\BIC{$R$}
\AXC{$Q$}
\RL{Rit$^2$}
\UIC{$Q$}
\RL{$\wedge_I$}
\BIC{$Q\wedge R$}
\RL{$\to_I^1$}
\UIC{$P\to Q\wedge R$}
\end{prooftree}
$$
Click or focus on the formula and press ENTER to start, Escape to quit.
- Try to express this in MathML
What about Diagrams?
- Automated generation of tactile graphics is challenging
- Advanced, bespoke content
- Generated with LaTeX packages
- Many Standard techniques do not work
Some examples from Abstract Algebra
Processing Labels
- Easy: Separate Text vs Math
- PreTeXt already does this for us: Text vs LaTeX
- Translate Text with Liblouis
- Translate Math with SRE
- Hard: Fitting tactile labels into graphic
- Hard: Understanding graphical content
Processing Labels
- Easy: Separate Text vs Math
- PreTeXt already does the for us: Text vs LaTeX
- Translate Text with Liblouis
- Translate Math with SRE
- Hard: Fitting labels into graphic
- Braille uses different space than text
- Labels are measured
- Bounding boxes are created and aligned
Workflow Example
- Creating Braille bounding boxes in the diagram

Workflow Example
- Fitting ASCII Braille into bounding boxes

Workflow Example
- Full rendering in Unicode Braille Font

Diagrams for Screenreading
- Once the graphics are understood we can do much more
- Diagrams can be handled via screen readers as well
- Goal is to have them all similar to our chemistry at progacc.com/chemistry
circle1group1group4group-2group-3group-4group-6group-7group-8group-9group-10group-11group-12group-13group-40group4group11circlegroup-2group-3group-2group-3group-21group1group-32group1group42circlegroup-6group-7group-8group-9group-10group-11group-12group-13group-4group-6group-7group-8group-9group-10group-11group-12group-13group-61group4group-72group4group-83group4group-94group4group-105group4group-116group4group-127group4group-138group4
Click or focus on the diagram and press A to start, Escape to quit, Escape to quit.
Talking to the Embosser
- The entire image is measured and scaled as much as possible to fit onto a page
- Good-looking image on the screen frequently does not look good embossed
- Result is usually suboptimal
- Scaling of graphics should not apply to Braille
- Embosser do not work well with SVG
- Conversion to PDF necessary
- Fonts need to be explicitly un-embedded
Example: Poorly embossed

- Diagram in PDF with fonts
Example: Legibly embossed

Example: Best solution

Some Conclusions
- Making Math accessible on the web is quite straightforward
- Make sure to work from sources
- Do not replace LaTeX with MathML!
- MathJax v4 is about to be released
- Currently Nemeth Braille:
- Adding other variants, e.g. UEB
- Definitely 8-dot LaTeX Braille
- Missing parts:
- Intelligent Linebreaking
- Form factors and Reflow
- Diagrams are really exciting... More tomorrow!